System and method for generating a 2D image from a tomosynthesis data set

ABSTRACT

A 2D mammogram image is synthesized from at least one of tomosynthesis projection images and/or the tomosynthesis reconstructed image data. In a simplest form, the mammogram may be synthesized by selecting one of the tomosynthesis projection images for display as a synthesized mammogram. Other methods of synthesizing a mammogram include re-projecting and filtering projection data and/or reconstructed data. The synthesized mammogram is advantageously displayed together with at least a portion of the reconstructed data to aid in review of the reconstructed data. The present invention thus provides a familiar image which may be used to facilitate review of a tomosynthesis data set.

RELATED APPLICATIONS

This patent application is a continuation-in-part and claims priorityunder 35 U.S.C. §120 to U.S. patent application Ser. No. 12/276,006filed Nov. 21, 2008 now U.S. Pat. No. 7,760,924 which is a continuationin part of U.S. patent application Ser. No. 11/827,909 Filed Jul. 13,2007 now U.S. Pat. No. 7,616,801, which is a continuation of applicationSer. No. 11/604,069 filed on Nov. 24, 2006, now abandoned, which is acontinuation-in-part of application Ser. No. 11/271,050, filed on Nov.10, 2005 now U.S. Pat. No. 7,577,282 and which is a continuation in partof application Ser. No. 10/305,480 filed on Nov. 27, 2002, now U.S. Pat.No. 7,123,684 and which is a continuation in part of application Ser.No. 10/723,486 filed Nov. 26, 2003 now U.S. Pat. No. 7,831,296. BothSer. No. 11/827,909 and Ser. No. 11/271,050 claim priority under 35U.S.C. §119 to provisional application 60/628,516 filed Nov. 15, 2004and application 60/631,296 filed Nov. 26, 2004.

FIELD

This patent specification pertains to x-ray mammography andtomosynthesis, and more specifically to techniques and equipment foracquiring and/or synthesizing, processing, storing and displayingmammograms, tomosynthesis projection images, synthesized two-dimensional(2D) images and tomosynthesis reconstructed images, and to medical imagesoftcopy reading systems, to hanging protocols and to other medicalimage display features.

BACKGROUND

Mammography has long been used to screen for breast cancer and otherabnormalities and for diagnostics. Traditionally, mammograms were formedon X-ray film, but more recently flat panel digital imagers have beenintroduced that acquire a mammogram in digital form and therebyfacilitate analysis and storage and provide other benefits as well.Further, X-ray tomosynthesis of the breast has been proposed recently,as discussed in the earlier-filed applications identified above, andclinical testing has been carried out. The assignee of this patentspecification, Hologic, Inc., has demonstrated at trade shows in thiscountry a fused, multimode mammography/tomosynthesis system that takeseither or both types of images, either while the breast remainsimmobilized or in different compressions of the breast.

Dedicated breast tomosynthesis systems also have been proposed. However,in clinical use it can be desirable for a number of reasons to assessboth tomosynthesis images and conventional mammograms of the patient'sbreast. For example, the decades of conventional mammograms have enabledmedical professionals to develop valuable expertise. Mammograms offergood visualization of micro-calcifications, and can offer higher spatialresolution when compared with tomosynthesis images. While tomosynthesisimages provided by dedicated breast tomosynthesis systems in the arthave other desirable characteristics (i.e., better visualization ofstructures), such systems do not leverage the existing interpretationexpertise of medical professionals. In addition, the increased mobilityof patient data and varied capabilities of imaging centers will requirethe ability to provide mechanisms that enable images to be displayedusing whatever resources are available at the imaging center, withoutregard to the original acquisition format of the image.

SUMMARY

Tomosynthesis as used in the systems and methods disclosed in thispatent specification typically involves acquiring a plurality oftomosynthesis projection images Tp at respective angles relative to thebreast, and reconstructing there from a plurality of tomosynthesisreconstructed images Tr representative of breast slices that haveselective thicknesses. According to one aspect of the invention, asynthesized 2D image is generated using at least one of thetomosynthesis projection images TP and/or the tomosynthesisreconstructed images Tr. The reconstructed images may be reconstructedusing any one of a variety of techniques, including but not limited tofiltered back projection in either spatial or frequency domain, maximumlikelihood reconstruction, iterative reconstruction, reconstructionusing algebraic methods, minimum likelihood or other known or developedthree dimensional reconstruction methods. The may be obtained usingprojection data obtained based on any coordinate system, including aCartesian coordinate system, a cone beam coordinate system, where thecone beam coordinate system may be defined by geometric information ofan associated tomosynthesis acquisition system or alternatively may be avirtualized cone beam coordinate system defined relative to a ‘virtual’tomosynthesis acquisition system. Further, following reconstruction, thereconstructed data may projected onto any other different coordinatesystem; for example, reconstruction data obtained using a firstcoordinate system may be projected into a second, different coordinatesystem. For example onto reconstructed data may be projected onto aplane of a different orientation, a cone beam reconstruction may beprojected onto a virtual cone beam coordinate system or a Cartesiancoordinate system, etc. In summary, projection images may bereconstructed onto any first coordinate system and then projected ontoany second, different coordinate system.

The synthesized 2 D image is referred to herein as a synthesizedmammogram (Ms) or other synthesized 2D tomosynthesis image (T2d). Thesynthesized 2D image may be generated using any combination oftomosynthesis projection data or reconstructed data generated usingfrom, or projected onto, any coordinate system. The synthesized 2D imageof the present invention is advantageously displayed together withtomosynthesis image data (Tr and/or Tp images) at a review workstation.With such an arrangement, a medical professional may utilize existingexpertise gained from past review of mammogram data to more efficientlyassess and view the 3D tomosynthesis data, without independentacquisition of a mammogram.

In another embodiment, the synthesized 2D image Ms may be displayedtogether with an Mp image previously obtained for the patient, to enablecomparison of like images using known methods before using the Tr data.Conversely, the method of synthesizing 2D images from Tp and/or Tr datamay be used to compare mammograms obtained by a mammography-only machineagainst existing tomosynthesis data for a patient, thereby increasingthe utility of tomosynthesis data by facilitating transport betweensystems of differing capabilities. Thus there are a variety of systems,including mammo only systems, tomo only systems and combo systems, whichmay benefit from the ability to synthesize a 2D image from tomosynthesisdata, either for comparison with mammography data, or increasing theefficiency of diagnostic workflow.

Proper display techniques make the presentation of Ms, Mp, Tp and/or Trimages (collectively referred to here as T images) more effective andefficient for review by health professionals. When tomosynthesisprojection images Tp are acquired, (with or without conventional 2Dmammograms Mp) improved display methods facilitate the display of both Tand Mp and/or Ms images.

Effective display approaches also are desirable when tomosynthesisimages Tp and/or Tr that are acquired at one time need to be compared tomammograms Mp and/or to tomosynthesis images Tp and/or Tr acquired at adifferent time. In situations where an Mp image is not available for aparticular time, but Tp and/or Tr images are available, the presentinvention enables generation of a synthesized mammogram image Ms.Effective displays also are desirable when only Tr and/or Tp images arebeing displayed.

An Ms image may be provided in any number of ways using one or more Tpimages and/or one or more Tr images. A variety of techniques forgenerating Ms images will be described in more detail below. The Msimage may be dynamically generated prior to display, or alternativelymay be pre-generated and stored. For example, Ms images may bedynamically synthesized prior to display of Tr/Tp images, may begenerated upon acquisition of the Tp images and stored with Tp images,or may be generated following reconstruction of the Tr images, using acombination of Tp and Tr images.

The display may be adapted to provide concurrent, toggled, overlaid orcine display of any combination of one or more of the Ms, Mp, Tp and Trimages. Concurrent display may be in the form of a side by side view,either on the same display or on neighboring displays, or alternativelymay be in the form of a thumbnail scout view of one image providedwithin another image. When viewing images concurrently, the presentinvention supports reflective marking of the different images; forexample should the technician mark an area of interest on the mammogram(or tomo slice) or move a marker on the mammogram, the mark and/ormovement of the mark is reflected in the appropriate location of thetomo slice (or mammogram).

Another display issue relates to Computer Aided Detection (CAD) methodsthat use computer analysis of images to identify locations and possiblyother characteristics of suspected abnormalities. CAD marks currentlyare placed on or are otherwise associated with mammogram images Mp, butit may be useful to place them at the appropriate location on Tr and/orTp images or to otherwise associate them with Tr/Tp images. Conversely,it may be desirable to obtain CAD marks by processing Tp and/or Trimages, and place them at appropriate locations on Mp images, oralternatively at appropriate locations in an Ms image.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating flow of data through a systemwhich includes a combination mammography/tomosynthesis acquisitionstation or a tomosynthesis only acquisition station and wherereconstruction of tomosynthesis slice images Tr and synthesis of the Msimages occurs after storage of acquired tomosynthesis projection imagesTp.

FIG. 2 is a block diagram illustrating flow of data through a systemwhich includes a combination mammography/tomosynthesis acquisitionstation or a tomosynthesis only acquisition station and where thereconstruction of images Tr occurs before storage of the image data.

FIG. 3 is a block diagram illustrating flow of data through a systemwhich includes a mammography-only acquisition system, and wherereconstruction of tomosynthesis slice images Tr and/or synthesis of theMs images occurs after storage of acquired tomosynthesis projectionimages Tp. FIG. 3 illustrates an example where four units acquiring Tpimages feed a single unit that reconstructs Tr images.

FIG. 4 illustrates a concurrent display of an Ms image and a Tr insubstantially same area on a screen, with an example of a non-numericindication of a thickness and position in the breast of a breast slicerepresented by a Tr image.

FIG. 5 illustrates a concurrent display of Ms and Mp images, at separateareas on a screen or as combined images.

FIG. 6A illustrates a display of a Tr image, with an integrated windowincluding a scout view of a 2D synthesized image, for use in guiding amedical professional's evaluation of the Tr data.

FIG. 6B illustrates a display of a Tr image with an integrated windowthat includes both an Ms image and a legacy Mp image, shown as scoutviews, for use in guiding a medical professional's evaluation andworkflow using the Tr image data.

FIG. 7 illustrates a display of Ms/Mp/Tr/Tp images with CAD marks and anon-numeric indication of Tr images in which CAD marks exist;

FIG. 8 illustrates a display of an MS image together with a Tr/TP image,wherein a region of interest marker placed in a first image is reflectedinto a second image;

FIG. 9 is a block diagram illustrating exemplary components of an x-rayacquisition and display system that incorporates the 2D synthesis methodof the present invention;

FIG. 10 illustrates image planes of a mammogram and tomosynthesis sliceimages;

FIGS. 11 a-11 c illustrate, respectively, tomosynthesis reconstructioninto a Cartesian geometry, into a cone-beam geometry and into a virtualgeometry; and

FIG. 12 is a flow diagram provided to illustrate exemplary steps thatmay be performed in a 2D image synthesis process which usestomosynthesis data.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

In describing preferred embodiments illustrated in the drawings,specific terminology is employed for the sake of clarity. However, thedisclosure of this patent specification is not intended to be limited tothe specific terminology so selected and it is to be understood thateach specific element includes all technical equivalents that operate ina similar manner.

The following abbreviations shall have the following definitionsthroughout this application. The notation Mp refers to a conventionalmammogram, which is a two-dimensional projection image of a breast andencompasses both a digital image as acquired by a flat panel detector oranother imaging device and the image after conventional processing toprepare it for display to a health professional or for storage, e.g. inthe PACS system of a hospital or another institution. The termMp_(current) refers to a mammogram that is acquired by an acquisitionsystem for patient diagnosis, while the term Mp_(legacy) refers to amammogram image of a patient that had been taken on a previous review ofthe patient, for example one that is to be used for comparison againstan Mp_(current) to identify changes in a patient's breast structure.

Tp refers to an image that is similarly two-dimensional but is taken ata respective tomosynthesis angle between the breast and the origin ofthe imaging X-rays (typically the focal spot of an X-ray tube), and alsoencompasses the image as acquired as well as the image after beingprocessed for display or for some other use. Tr refers to an image thatis reconstructed from images Tp, for example in the manner described insaid earlier-filed patent applications, and represents a slice of thebreast as it would appear in a projection X-ray image of that slice atany desired angle, not only at an angle used for Tp or Mp images.

The term Ms refers to synthesized 2D projection images which simulatemammography images, such as a craniocaudal (CC)_ or mediolateral oblique(MLO) images, and are constructed using tomosynthesis projection imagesTp, tomosynthesis reconstructed images Tr or a combination thereof. Msimages may be provided for display to a health professional or forstorage in the PACS system of a hospital or another institution.

The terms Tp, Tr, Ms and Mp also encompasses information, in whateverform, that is sufficient to describe such an image for display, furtherprocessing, or storage. The images Mp, Ms. Tp and Tr typically are indigital form before being displayed, and are defined by informationidentifying properties of each pixel in a two-dimensional array ofpixels. The pixel values typically relate to respective measured orestimated or computed responses to X-rays of corresponding volumes inthe breast (voxels or columns of tissue). In a preferred embodiment, thegeometry of the tomosynthesis images (Tr and Tp) and mammography images(Ms, Mp) are matched to a common coordinate system as described in U.S.patent application Ser. No. 11/667,650 “Matching Geometry Generation andDisplay of Mammograms and Tomosynthesis Images”, filed Nov. 15, 2005 andincorporated herein by reference.

FIG. 1 illustrates flow of data in one example of an image generationand display system disclosed in this patent specification. An image dataacquisition system 1 acquires tomosynthesis image data for Tp images ofpatients' breasts, and can take the form of and use the acquisitionmethods of any of the systems disclosed in said earlier-filed patentapplications. If the system is a combo system, Mp images may also begenerated (as indicated by dashed line and label Mp_(current) in FIG.1). Some dedicated tomosynthesis systems or combo systems may be adaptedto accept and store legacy mammogram images (indicated via dashed lineand legend Mp_(legacy) in FIG. 1) in a Picture Archiving andCommunication System (PACS) storage device 2, although it is not arequirement that any Mp images be acquired or pre-stored.

Following tomosynthesis image acquisition, the data describingprojection images Tp are sent to storage device 2, which is preferably aDICOM-compliant PACS. When images are needed for display 5, the data Tpimages are sent, from either acquisition system 1 or from storage device2, to a computer system 3 configured as a reconstruction engine that canperform tomosynthesis reconstruction into images Tr representing breastslices of selected thickness and at selected orientations, as disclosedin said earlier-filed patent applications and detailed below. Thecomputer system may be further configured with 2D synthesisfunctionality 4, which may operate substantially in parallel withreconstruction engine 3 to generate a synthesized 2D image(interchangeably referenced as T2d or Ms). The reconstructed sliceimages Tr are then sent to a display system 5 so that they can beviewed. If the reconstruction engine 3 is connected to display 5 via afast link, then large datasets can be transmitted quickly. Other images,such as the Ms, Mp and/or Tp images may also be forwarded to the displayunit for concurrent or toggled viewing.

Over time, there will likely be improvements to the acquisition systemsand to the display systems, which may result in hardware and softwareupgrades and changes to reconstruction algorithms. This can createissues in viewing images taken previously. It may be important to beable to recall from storage and reconstruct an image that looksidentical (or is at least comparable) to the way it looked when it wasreconstructed and displayed in the past, or vice versa.

Consider the example where an improvement in reconstruction algorithmsimproves image quality so as to allow detection of a cancerous lesion inan image where it was not visible using a previous version of thereconstruction algorithm and the then existing standard of care. Whileit could be useful to see older images processed with the neweralgorithms, it may also be important to allow the re-display of imagesas they were viewed during an original detection/diagnosis. One way toaccomplish this in accordance with the disclosure in this patentspecification is to put a version number or some other information inthe data for Tp images, which identifies the software and/or hardwareversions of the Tp image data acquisition and/or Tr image reconstructionsystem at the time of acquisition, or to otherwise associate suchinformation with the Tp images. During reconstruction at a later time,the reconstruction engine reads this version number or other similarinformation and reconstructs using the appropriate algorithm. Thus,system upgrades can maintain a library of older algorithms and/orhardware so as to be able to reconstruct using the proper technique.

In addition it may be desirable to port existing images, generated usingsystems having different capabilities, to new systems. For example apatient may have compiled a history of mammogram images associated withpast screenings. Such a patient may be examined at a facility withupdated equipment, for example one that includes a dedicatedtomosynthesis system. To compare historical information against existingdiagnostic images it may be desirable to store such legacy Mp images,generate Ms images from a tomosynthesis acquisition of the dedicatedtomo system and compare like-formatted representations.

Therefore, an alternative design of an image acquisition and displaysystem of the present invention is illustrated in FIG. 2. In thisexample, the reconstruction unit 3 and 2D synthesizer 4 are directlycoupled to the acquisition station 1, and it is the reconstructionimages Tr and synthesized images Ms that are sent to storage system 2for subsequent display on display devices 5, which may also store legacyMp images. One advantage of the configuration of FIG. 2 is in the way ithandles acquisition and synthesizing upgrades—if a new hardware/softwareversion has a modified reconstruction algorithm, then all Tr images andMs images reconstructed or synthesized from Tp image data taken afterthe upgrade will automatically reflect this new algorithm, and Tr and Msimages reconstructed or synthesized from Tp image data taken prior tothe upgrade will have been reconstructed with the older version andproperly stored as such. The images stored on a PACS will be the same asthey were viewed by the radiologist or other health professional duringthe detection/diagnosis or other earlier review. Another advantage ofthe system of FIG. 2 is the reduced system reconstruction burdencompared to the system in FIG. 1, where the reconstruction engine isjust prior to the display. If there are multiple acquisition systems,for example four systems that are all pushing images to the display,then the reconstruction engine will need to reconstruct images at 4times the rate of a reconstruction engine in a system having only oneacquisition system, for the same total patient throughput.

FIG. 3 illustrates another image acquisition and display system whichmay benefit from the 2D image synthesizing process of the presentinvention. In FIG. 3, the image acquisition device 11 is a mammographyonly device. One advantage of digital imaging is the portability ofdata; it is conceivable that patients that switch between differentimaging locations may be exposed to imaging equipment with differentcapabilities. For example, a patient may undergo an exam at an imagingcenter that includes a tomosynthesis only system, and subsequentlyundergo an exam at a location that includes a mammography onlyacquisition system, (or visa versa). In order for the medicalprofessional to easily compare images to identify changes in the breaststructure it would be desirable to synthesize a 2D mammogram fromexisting Tr and/or Tp data. In FIG. 3, storage device 2 is adapted tostore both legacy Mp (if any) as well as legacy Tp images. Dependingupon capabilities of the display device, (i.e., whether tomosynthesisdata can be viewed), the system may include reconstruction unit 3. Othersystems which have only the capability of viewing mammograms may notinclude this unit, and thus the unit and tomosynthesis data are allshown in dashed lines in FIG. 3. In the system of FIG. 3, when amammogram Mp is acquired, it is either stored in storage device 2 orforwarded to display 5. Tp and/or Tr data are retrieved from the storagedevice and forwarded to the 2D synthesizer 4. The resulting Ms image isdisplayed together with the current Mp on display 5. It is envisionedthat in such systems the 2D synthesizing software may be provided as adownloadable application that facilitates viewing of tomosynthesis dataon existing mammography systems.

The question of which system design will place a greater burden on thePACS storage of an institution will depend upon the sizes of the rawprojections Tp and of the reconstructed images Tr. In general, if theraw projections Tp are smaller than the reconstructed images Tr, itmight be desirable to save in PACS the raw or preliminarily processeddata for Tp images and reconstruct the final Tr images on demand fordisplay or other use. In all cases it may be desirable to keep both ofthese sizes as small as possible.

One way to reduce the size of an original dataset for a Tp image is tobin the projection Tp data to as large a pixel size as practical withoutreducing clinical efficacy of the final Ms, Tp or Tr images. Methodsthat can be used to reduce the Tp image size are described in U.S.patent application Ser. No. 11/271,050, (referred to herein as the '050application) filed Nov. 10, 2005 by the assignee of the presentinvention, and incorporated by reference herein.

For storage, transmission to remote locations, and/or other purposes,the images can be formatted consistent with DICOM standards. Forexample, each raw or displayed projection image set, synthesized imageor reconstructed slice image set for a single view is stored as a singleSecondary Capture image instance according to DICOM. The image pixeldata can be encoded in a selected compressed format (CODEC) thatincludes all projection or slice images.

As shown in FIGS. 1-3, the imaging and display system of the presentinvention includes a 2D synthesizer for generating 2D images simulatingmammograms taken at both a CC and MLO orientation using a combination ofone or more Tp and/or Tr images. A display of the system preferablyshould be able to display Ms, Mp and Tr (and/or Tp) images concurrently(either in separate windows on the display, on separate monitors of atechnology workstation, or overlaid) or sequentially or in toggled mode,wherein the Ms, Mp, Tp and Tr images may be those currently acquired, orthose that were acquired in previous studies. Thus, in general, thedisplay can simultaneously or sequentially or in toggled mode displaymammograms (Ms, Mp) and tomosynthesis images Tr (and/or Tp) from thecurrent and previous studies. Tr slices can be reconstructed all to thesame size, which can be the same as the size of an Mp or Ms image of thebreast, or they can be initially reconstructed to sizes determined bythe fan shape of the x-ray beam used in the acquisition and laterconverted to that same size by appropriate interpolate]on/extrapolation.

Images of different types and from different sources can be displayed indesirable size and resolution. For example, an image can be displayed in(1) Fit To View Port mode, in which the size of the displayed image sizeis maximized such that the entire imaged breast tissue is visible, (2)True Size mode, in which a display pixel on the screen corresponds to apixel of the image, or (3) Right Size mode, in which the size of adisplayed image is adjusted so that it matches that of another imagethat is concurrently displayed or with which the displayed image is orcan be toggled. For example, if two images of the same breast are takenand are not the same size or do not have the same special resolution,provisions are made to selectively zoom in or zoom out one of them, orzoom both, such that they appear to be the same size on the screen whenthey are concurrently displayed or the user toggles between them, tofacilitate comparison or to otherwise facilitate detection/diagnosis.Known interpolation/extrapolation and weighting techniques can be usedin such re-sizing, and known image processing technology can be used tomake other characteristics of the displayed images similar in a way thatfacilitates detection/diagnosis.

Selected hanging protocols are provided that are unique to the differenttypes of images with which the disclosed system deals. As one example,the hanging protocols for 2D images (e.g. Ms or Mp images) and 3D images(e.g. Tr images) are linked so that when one type of image is displayedfor a given breast the other type is displayed as well. For example,when the Ms/Mp image of a breast is displayed, a tile of the Tr imagesand/or of the Tp images is automatically displayed at the same time,with a desired hanging protocol that may involve scrolling or cine modepresentation, or may require user input so select a particular subset ofthe Tr and/or Tp images or a particular individual Tr/Tp image. Thus, acombined hanging protocol set can be provided for 2D and 3D images thatare concurrently displayed (either on a common display on adjacentdisplays, or overlaid) or toggled such that only one type is displayedat one time. In addition, the combined hanging protocol can includeprovisions for linked display of CAD information associated with one orboth of the 2D and 3D images involved in the hanging protocol.Alternatively, the hanging protocols for 2D images are made differentfrom those for 3D images. Methods of identifying which image correspondsto which image type in displays of Ms, Mp, Tr and/or Tp images aredesirable. One example of such a method is illustrated in FIG. 4. Anicon is used to identify an image type. In this non-limiting example,the symbol MS on the left image 300 indicates that it is a synthesizedmammogram. The symbol T on the right image 310 indicates that it is atomosynthesis slice image Tr. Similarly, a symbol Tp (not shown) can beused to indicate that the displayed image is a tomosynthesis projectionimage Tp, a symbol 2D may be used to indicate that it is a 2D image, andthe symbol 3D (also not shown) can be used to indicate that an image onthe display is a 3D image. Other symbols/icons serving a similar purposecan be used instead of, or in addition, to those identified above. Inthe alternative, the images can be displayed without an identificationof the type of image. For example, a Tr image and an Mp or Ms image canbe displayed at the same time or toggled without displaying anindication of the type of the image that is visible. This may bedesirable in cases such as when a user has a familiar hanging protocoland does not need an express identification of the type of image.

The system described as a non-limiting example in this patentspecification is capable of receiving and displaying selectively thetomosynthesis projection images Tp, the tomosynthesis reconstructionimages Tr, the synthesized mammogram image Ms and/or the mammogramimages Mp, or a single type, or any sub combination of types. It canreceive images stored uncompressed, losslessly compressed, and lossylycompressed. It can also include algorithms to decompress images sent incompressed format. The system has software to perform reconstruction oftomosynthesis image data for images Tp into images Tr and software forsynthesizing mammogram images Ms. Further, it can include software togenerate 3D display images from the tomosynthesis reconstructed imagesTr using standard known methods such as MIP (Maximum IntensityProjection), summing, and/or weighted summing algorithms. FIG. 5illustrates an exemplary display of an Mp image together with a 2Dsynthesized image; each image may be labeled to indicate whether theimage is from a current acquisition, or based on legacy data. Forexample the Mp image may be a stored legacy mammogram, and the 2D Msimage may be generated from a current tomosynthesis acquisition and maybe provided as an initial view to guide the medical professional'sperusal of the tomosynthesis data. Alternatively, the Mp image may bebased on a current acquisition, for example via a combo mammo/tomosystem or by a mammography-only system, and the Ms image may begenerated from previously stored tomosynthesis data, such as describedin FIG. 3, thereby allowing for comparison of like images to more easilyidentify regions of interest.

FIG. 6A illustrates the display of the synthesized 2D image as a scoutview thumbnail image which may be provided as an overlay in a display ofTp and/or Tr images. Such a scout view may be used to guide the workflowof the medical professional during review of the tomosynthesis images.FIG. 6B illustrates two scout views, showing, for example, a legacy 2Dimage together with a current 2D image. The ability to compare the twoimages may further assist the medical workflow. Although FIG. 6Billustrates a legacy Mp image together with an Ms image, any two 2Dimages associated with differently timed acquisitions could be used, andthe present invention is not limited that the particular images shown inthe figures.

FIG. 7 illustrates a display of two synthesized 2D images, of differentviews (CC and MLO). In the embodiment of FIG. 7, a Computer AssistedDetection (CAD) process has been applied to either the synthesized 2Dviews or alternatively to the reconstructed tomosynthesis data,providing resulting CAD marks 350. As described in the '050 application,the CAD marks resulting from processing a mammogram may be projectedonto the 3D tomosynthesis image and visa versa. The present inventionfurther envisions that the CAD marks may be similarly translated acrossimages when using 2D synthesized images.

A variety of methods can be used to select the particular image to bedisplayed. For example a menu driven interface may be automaticallypopulated with the types of images that are available for display,including both currently acquired images and a selection of availablelegacy images. Software allows the selection of one or more imageplanes, for use in image processing, or to change window/level or tochange slice height, etc. The menu driven interface may be furtherpopulated with display arrangements, including overlaid, cine, insetviews, etc. Alternative methods, such as drag and drop techniques can beused to position images on the screen. These sets of images can be onone monitor, or on multiple monitors or other displays.

When more than one image is displayed, it can be convenient to have theimages all be displayed at the same pixel spacing, using knowninterpolation or extrapolation methods applied to digital images. Thiscan facilitate image comparison. As an example, if the prior mammogramwas acquired on a system using 100 micron pixel spacing, but the currentmammogram was acquired on a system using 70 micron pixel spacing, thedisplay can map the images so the pixel spacings are identical. Thispixel spacing adjustment can also be used for Ms, Mp and Tr/Tp images.In a preferred embodiment, (with the exception of thumbnail scout viewssuch as those of FIGS. 6A and 6B) the Ms, Mp and Tr/Tp images aredisplayed at the same pixel size. This is especially useful inperforming overlaid or toggled image display, with the Ms, Mp and Tr/Tpimages on top of each other. Thus, an object in a Tr image will appearat the same place as in the corresponding Ms/Mp image. If the two imagesare not at the same pixel size, toggling between them may show adistracting change due to the difference in pixel size. Matching thepixel spacings for all images on the display is only one possibility. Acapability to change the pixel spacings of any image or sets of images,such as would occur when one zoomed a region of a breast, can also beincluded.

Zooming can be done on any of the images on the display. For example, ina combo overlay display mode, the zoomed area will zoom both the Ms/Mpand the Tr slice images as they are toggled. In other words, no matterwhat image type is displayed, it will be zoomed. Window/level can beindependently, or jointly, applied to any combination of images on thedisplay. In particular, for the Tr images the window/level can beapplied to just the single displayed Tr slice image, or all the Tr sliceimages. If there is a magnified region of an image, window/level can beselectively applied just to the magnified region or to the entire image.

According to another aspect of the invention the synthesized mammogrammay be used in conjunction with a tomosynthesis image at thetechnologist workstation to assist in the identification oftomosynthesis slices associated with regions of interest. For example,referring now to FIG. 8, a technologist or other user may place a marker(410A) or otherwise highlight regions of interest (420A, 430A) on amammogram. The present invention automatically reflects the marker, bygenerating a duplicate marker at a corresponding x-y location on thetomographic image 400B as markers 4108, 420B, and 430B. As a user movesthe marker do different locations within the image (for example, using amouse to drag a cursor associated with the mark or region), thereflected mark moves similarly in the alternate image. The automaticreflection of the marker can easily be accomplished using standardgraphic imaging techniques because the coordinate systems of the twoimages have already been aligned for display purposes; thus there willbe a one to one relationship between the x-y location of the added mark,and the x-y location on the alternate image where the mark should bereflected. The makers essentially lie on a two-dimensional plane whichlies over the respective images, and remain in position as the userscrolls depthwise through the tomographic slice images. Thus the markerwill remain over the tomosynthesis slices as the user scroll depth-wisethrough the tomosynthesis slices. Similarly, the technician may placemarkers on the tomosynthesis image which are reflected onto themammogram. With such an arrangement the technician is able to quicklydiscern how an artifact in the tomosynthesis slice appears in amammogram image.

FIG. 9 illustrates an overall mammography/tomography system in which thepreferred but non-limiting examples discussed above can be implemented.The Figure illustrates in block diagram form an x-ray data acquisitionunit 100 that includes an x-ray source 110 imaging a breast 112. Anx-ray imager 116 such as a flat panel x-ray imager commerciallyavailable from the assignee of this patent specification generatesprojection image data that can be a mammogram Mp or a tomosynthesisprojection image Tp. X-ray source 110 is mounted for movement so thatimages Tp can be taken at different angles. X-ray imager 116 can bestationary or it can also move, preferably in synchronism with movementof x-ray source 110. Elements 110 and 116 communicate with x-ray dataacquisition control 118 that controls operations in a manner known fromsaid earlier-filed patent specifications. X-ray image data from imager116 is delivered to processing unit 120. Processing unit 120 comprisesreconstruction software 122, which may be stored in a computer readablemedium of unit 12. The reconstruction software processes x-ray imagedata as known from said earlier-filed patent application into Tp and Trimage data, which may be stored in storage device 130 and displayed atimage display unit 150 as disclosed in the various embodiments describedabove.

In particular tomosynthesis reconstruction may use any of the methodsdescribed in the “Matching Geometry” patent application (Ser. No.11/667,650) referenced above. The disclosed process and system generateand display tomosynthesis slice images of a patient's breast such thatan object in the breast is at same or at least matching relative placesin each slice image in which it is visible and, preferably, also at thesame or at least matching place as in a conventional mammogram of thesame breast. To achieve this, the method and system described in the“Matching Geometry” patent application obtain 2D x-ray projection datafor tomosynthesis images preferably using a cone-shaped orpyramid-shaped imaging x-ray beam, and generate tomosynthesis imagessuch that they conform to the same geometric coordinate system as amammogram and, preferably, to the same coordinate system as a 2Dprojection mammogram. As a result, anatomical structures appear atgeometrically matching or corresponding places in such tomosynthesisimages and, preferably, in the mammogram. In one embodiment, thetomosynthesis images can be generated in a two-step computer-implementedprocess that first reconstructs tomosynthesis images in an initialcoordinate system, such as a Cartesian coordinate system, in whichobjects are not or may not be at matching positions in differenttomosynthesis images or in the mammogram, and then projects those imagesinto another coordinate system, such as the coordinate system of themammogram. In a second embodiment, the reconstruction can directlygenerate tomosynthesis images in a desired coordinate system, e.g., thecone beam geometry of the mammogram. According to one aspect of theinvention, this direct reconstruction can further be projected onto adifferent coordinate system if desired, for example to modify the planeof reconstruction.

For example, FIG. 10 illustrates a front view where the long axis of thecompressed breast 10 is normal to the sheet. The breast 10 is compressedbetween an image receptor 12, such as a flat panel digital imager, and acompression paddle 14, and is imaged with a cone-shaped orpyramid-shaped x-ray beam 16 from an x-ray source 18. Two objects areillustrated in breast 10, object #1 at slice A and object #2 at slice B.The term object is used here to refer to any structure that can beimaged in a mammogram or a tomosynthesis image, such as a lesion in thebreast, and the term slice is used to refer to a layer of the breast ofa finite thickness, e.g. thickness in the z—direction, that is less thanthe total breast thickness. For example, a slice can be a few mm thick,or thinner or thicker. Because the objects #1 and #2 are along the samex-ray trajectory 20, they appear superimposed in a mammogram. However,because x-ray trajectory 20 is not normal to the image plane of receptor12, as is the general case with x-ray trajectories when using suchcone-beam geometry, the two objects appear at different xy locations intomosynthesis projection images for slice A and slice B. The “MatchingGeometries” application describes a reconstruction and display methodfor tomosynthesis images that matches the coordinates of tomosynthesisimages with mammograms. All relevant x-ray measurements can be obtainedas respective 2D sets of pixel values (x-ray measurements for elementalpicture areas) at each of several different angles of an x-ray beamrelative to a breast, e.g. at several equidistant angles over a range.Other raw x-ray data acquisition techniques can be used in thealternative. After pre-processing of the type known in the mammographyand breast tomosynthesis art, those pixel values can be reconstructedinto a rectangular Cartesian coordinate system (30 in FIG. 11 a) usingknown reconstruction algorithms such as filtered back projection,iterative reconstruction, maximum likelihood reconstruction, or others,for example as taught in said patent application Ser. No. 10/723,486,incorporated herein by reference. As illustrated in FIG. 11 a, thevoxels (elemental volume elements) that are imaged as respective pixelsin the tomosynthesis slice images are aligned along lines normal to theimage plane of receptor 12. The result can be conceptualized as a set ofpixel values representing x-ray properties of the voxels that are in the3D space bound by the image plane of receptor 12 at the bottom,compression paddle 14 on top, and on the sides by the boundaries of anx-ray beam 16 that impinges on receptor 12, and are uniformly spaced inxy planes. Because the x-ray beam 16 is cone-shaped, the sides of this3D space slope at least on three sides of the beam, and the x-raytrajectories from source 18 to receptor 12 diverge in the general case.Thus, in the general case each x-ray trajectory such as trajectory 20 isnon-normal to the image plane of receptor 12. Tomosynthesis image slicesthat match the geometry of the mammogram can be obtained by projectingeach of several horizontal breast slices separately onto the image planeof the mammography images, taken along the actual x-ray trajectoriesincluded in x-ray beam, using a computer-implemented process adaptedwithout undue experimentation to a particular-x-ray data acquisitiongeometry by a programmer of ordinary skill in the art.

Alternatively, for a cone beam x-ray illumination, the reconstructiongeometry can be a cone beam coordinate system 31 shown in FIG. 11 b,where the voxels that correspond to pixels in the tomosynthesis sliceimages are at different xy spacings (and differ in size at least in thexy plane) in different slices and corresponding voxels of differentslices are along the same (generally sloping) x-ray trajectory. For thereconstruction processing carried out by reconstruction software 122, ageometry matrix can be defined from a geometry calibration file andinput projection angles appropriate to the acquisition system 11 for usein backprojection, from fits to the matrix elements determined from ageometry calibration of the acquisition system 11 and input projectionangles measured by an encoder in the acquisition system 11. Imageprocessing and filtering can be carried out on the images prior toreconstruction, using image processing techniques known in technologiessuch as CT scanning and tomosynthesis. A known skin line correction canalso be applied at this stage. A backprojection can then be carried out,one tomosynthesis slice at a time, using the geometry matrix as follows:

$\begin{matrix}{{\begin{pmatrix}u \\v \\s\end{pmatrix} = {\left( M_{i} \right)\begin{pmatrix}x \\y \\z \\1\end{pmatrix}}}{d_{x} = {u/s}}{d_{y} = {v/s}}} & {{{Equation}\mspace{25mu} I}\;}\end{matrix}$

Where u, v, z is the location of the reconstructed pixel, Mi is the 3×4geometry matrix for projection I, (x,y,z) is the location of an imagepixel, and (dx, dy) is the location on the x-ray detector element orarea for the line that connects a focal spot in source 18 and the imagepixel.

It is important to note that the geometry matrix (M) is not limited to aCartesian geometry, or to the acquisition system geometry. Rather thepresent invention realizes that there may be advantages toreconstructing the data according to the geometry of any coordinatesystem. FIG. 11 c illustrates exemplary virtual cone beam geometry,where the cone beam coordinate system is derived from a hypotheticalacquisition system having virtual focal spot positioning. Such acoordinate system, like that of FIG. 11 b, has different pixel spacingfor each tomosynthesis slice. However, in such an arrangement the angleof pixel spacing close to the chest wall is increased. Other virtualcoordinate systems, associated with different hypothetical acquisitionsystem or which focus on different portions of the imaging area may besubstituted readily herein within the scope of the present invention.

The above reconstruction methods use a filtered backprojection processto project data to a known geometry. However, other known methods ofreconstruction can be used to achieve the same results, including butnot limited to iterative reconstruction, maximum likelihoodreconstruction, or others, for example as taught in said patentapplication Ser. No. 10/723,486.

The tomosynthesis image slices to be reconstructed can be parallel to a“default” reference plane as suggested by Equation 1 above.Alternatively, they can be at other desired orientations, defined by a4×4 matrix multiplication operation applied to the original 3×4 matrixM, according to:

$\begin{matrix}{{{\begin{pmatrix}u \\v \\s\end{pmatrix} = {\left( M_{i} \right)\begin{pmatrix}R_{3{x3}} & T_{3} \\O_{3}^{T} & I_{1}\end{pmatrix}\begin{pmatrix}x^{\prime} \\y^{\prime} \\z^{\prime} \\1\end{pmatrix}}}\mspace{20mu}{where}\mspace{11mu}\;{\begin{pmatrix}x \\y \\z \\1\end{pmatrix} = {\begin{pmatrix}R_{3{x3}} & T_{3} \\O_{3}^{T} & I_{1}\end{pmatrix}\begin{pmatrix}x^{\prime} \\y^{\prime} \\z^{\prime} \\1\end{pmatrix}}}}\mspace{11mu}} & {{Equation}\mspace{25mu}{II}}\end{matrix}$

For example, a preferred orientation can be an orientation in which aparticular mammogram is taken. Alternatively the use of the perspectivematrix allows the reconstructed image to be viewed at any orientation.For example, it may be desirable to have several sets of reconstructeddata which are related to a common set of projection images, but arereconstructed using different perspectives and coordinate systems.

In summary, reconstructing tomosynthesis slice images can involve: 1.)The selection of the orientation of image slices to be reconstructed.The slice can be either parallel to the “default” reference plane assuggested by Equation I, or at another more preferred orientation, whichis defined by a 4×4 matrix multiplication operation to the original 3×4matrix M, as expressed by Equation II; and 2.) Selection of thereconstruction voxel grid in space, which can be either a Cartesian grid(FIG. 11 a) or a Cone beam grid (FIG. 11 b) or virtual grid (FIG. 11 c).

Processing unit 120 further includes 2D synthesis software which usesone or more of the Tp and/or Tr images to synthesize a 2D image.

There are varieties of methods that can be used to synthesize a 2D imageusing tomosynthesis data. In a simplest form, any Tp image taken duringthe tomosynthesis scan may be used as 2D image. Tp images may be usedindividually, or alternatively a subset of Tp images or a subset of Trslices (reconstructed using any of the methods above) may be combined,using algebraic methods (averaging, weighted averaging, statistics orother methods), maximum intensity projection, or other known means toprovide the 2D image. One example of a method for synthesizing a 2Dimage will now be described although it should be understood that thepresent invention is not limited to any particular method ofsynthesizing a 2D image, but rather encompasses any synthesizingtechnique which can be used to generate a 2D image from a tomosynthesisdata set.

In an exemplary embodiment, a tomo data set consists of Tp0 rawprojections, Tp processed projections, and Tr reconstructed slices. TheTp processed projections have been processed as described in the '650application to perform at least one of coordinate geometry matching anddata set size reduction.

A single ‘synthesized’ 2D image T2d, analogous to the conventionalmammography image Mp, is built from the 3D tomo data set alone. Asdescribed above, the 2D synthesized image may provide a quick overviewof the breast anatomy to facilitate diagnosis and help the radiologistfocus on specific regions when analyzing the 3D slices. When reviewingimages on the display workstation, the image T2d may replace the Mpimage that would normally be present in a combo mode procedure, or maybe viewed against legacy Mp images, or displayed in a variety of othercombinations.

FIG. 12 is a flow diagram that illustrates exemplary steps that may beperformed in a 2D synthesis process. FIG. 9 assumes that methodsdescribed in the '650 application have been used to generate a set ofslices Tr as in put to the process, wherein the set of slices arerepresented in a Cone Beam or Cartesian coordinate system, oralternatively represented in a virtual coordinate system (associatedwith a virtual acquisition system). It should be understood thatalthough the below equations describe a method using Tr images, similarprocessing may be performed with any subset of the Tp images.

At step 910, the Tr data set is apportioned into a slabbed set of slicesTslab. That is a number of images Tr are effectively combined, usingmaximum intensity projection (MIP) or averaging to generate a set ofTslab slices. Equation III below illustrates how the set Tslab is formedusing MIP, while Equation IV below illustrates how the set Tslab may beformed using averaging.

Let a slice in the original set be Tr [j, z] where j is the pixel indexof the image and z is the slice number.Tslab[j,z]=MAX(Tr[j,z−Nslab/2], Tr[j,z−Nslab/2+1, . . . ,Tr[j,z+Nslab/2])  Equation III:Tslab[j,z]=AVE(Tr[j,z−Nslab/2], Tr[j,z−Nslab/2+1, . . . ,Tr[j,z+Nslab/2])  Equation IV:

At step 920, once the voxel values in the Tslab slices have beenselected, the set of slices is re-projected to produce an initial imageT2d0. Re-projection methods are well known in the field of imageprocessing. A source point and image plane is chosen, on opposite sidesof the image volume. Pixels are obtained by projecting the source pointthrough the slice set to an image plane point. The pixel value is summedat each slice location by interpolating values in the original slices.Note, in the case of cone beam coordinate system reconstruction asdescribed in the '650 application and above, the re-projection is just asum of pixel values, with no interpolation involved, and is representedby Equation V below:

$\begin{matrix}{{{Trep}\left\lbrack {j,z} \right\rbrack} = {{1/N}{\sum\limits_{z = {zmin}}^{zmax}\;{{Tslab}\left\lbrack {j,z} \right\rbrack}}}} & {{Equation}\mspace{20mu} V}\end{matrix}$

Where zmin, zmax may be chosen to exclude slices near the breastboundary, or skin. This may reduce artifacts. N=zmax−zmin+1.

Step 930 performs an optional step of filtering the re-projected imageto produce T2d. The filtering that is performed should be generally inthe direction of the source motion in the original tomosynthesis imageacquisition. Although not required, filtering may help reduce additionalblur produced in the re-projection due to artifacts in the slices Tr. Itis further noted that the filtering step 930 may be performed prior tothe re-projection of step 920, but at a computational cost.

The advantages of using the cone beam geometry reconstructed slices asinput are as follows. Interpolation at step 920 is simplified becausegeometric correlation was already performed in the reconstruction. Thusthe final image T2d will be registered geometrically with the originalset of slices Tr, as described in the '650 application. The registrationwould facilitate diagnosis as well as the display of CAD results on T2d,where the CAD results are derived from the 3D images Tr. The 3D CADresults may also be re-projected (or summed) as in step 2 and overlaidon T2d.

Accordingly several systems for displaying x-ray images together with 2Dimages that are synthesized from tomosynthesis image data have beenshown and described. The synthesized images may be generated anddisplayed in conjunction with combination mammography/tomographyacquisition stations and tomosynthesis only acquisition stations. Theimages may even be generated and displayed in combination withmammography only acquisition stations when legacy tomosynthesis data isavailable. With such an arrangement diagnostic efficiency is increasedbecause the provision of familiar images along with the tomosynthesisdata allows historical imaging expertise to be leveraged.

Having described exemplary embodiments, it can be appreciated that theexamples described above are only illustrative and that other examplesalso are encompassed within the scope of the appended claims. It shouldalso be clear that, as noted above, techniques from known imageprocessing and display methods such as post-production of TV images andpicture manipulation by software such as Photoshop from Adobe, can beused to implement details of the processes described above. The abovespecific embodiments are illustrative, and many variations can beintroduced on these embodiments without departing from the spirit of thedisclosure or from the scope of the appended claims. For example,elements and/or features of different illustrative embodiments may becombined with each other and/or substituted for each other within thescope of this disclosure and appended claims.

The invention claimed is:
 1. A method comprising: obtaining a pluralityof x-ray tomosynthesis projection images of a patient's breast;reconstructing the plurality of x-ray tomosynthesis projection imagesinto a three dimensional reconstructed image comprising a plurality ofslices; and synthesizing a two-dimensional mammogram using at least asubset of the three-dimensional reconstructed image, wherein the step ofsynthesizing includes projecting at least one tomosynthesis slice imagehaving a pixel spacing that differs from a pixel spacing of a mammogramonto a mammogram coordinate system; and displaying the synthesizedmammogram on a display device.
 2. The method of claim 1 including thestep of filtering at least one of the x-ray tomosynthesis projectionimages prior to the step of synthesizing.
 3. The method of claim 1wherein the step of reconstructing the plurality of tomosynthesisprojection images reconstructs the tomosynthesis projection images ontoa first coordinate system to provide an initial reconstruction.
 4. Themethod of claim 3 including the step of wherein the initialreconstruction is the three-dimensional reconstructed image.
 5. Themethod of claim 3 wherein the first coordinate system is selected from agroup including a Cartesian coordinate system, a cone beam coordinatesystem and a coordinate system associated with a virtual acquisitionsystem.
 6. The method of claim 3 wherein the step of reconstructing theplurality of tomosynthesis projection images includes the stepprojecting the initial reconstruction onto a second, differentcoordinate system to provide the three-dimensional reconstructed image.7. The method of claim 6 wherein the second coordinate system isselected from a group including a Cartesian coordinate system, a conebeam coordinate system and a coordinate system associated with a virtualacquisition system.
 8. The method of claim 3 wherein the step ofreconstructing includes the step of orienting said three-dimensionalreconstructed tomosynthesis image to a selected plane.
 9. A methodcomprising: obtaining a plurality of x-ray tomosynthesis projectionimages of a patient's breast; reconstructing the plurality of x-raytomosynthesis projection images into a three dimensional reconstructedimage comprising a plurality of slices, wherein the step ofreconstructing the plurality of tomosynthesis projection imagesreconstructs the tomosynthesis projection images onto a first coordinatesystem to provide a three-dimensional reconstructed image and whereinthe first coordinate system is selected from a group including aCartesian coordinate system, a cone beam coordinate system and acoordinate system associated with a virtual acquisition system;synthesizing a two-dimensional mammogram using at least a subset of thethree-dimensional reconstructed image; and displaying the synthesizedmammogram on a display device.
 10. The method of claim 9 wherein thestep of reconstructing the plurality of tomosynthesis projection imagesincludes the step projecting the initial reconstruction onto a second,different coordinate system to provide the three-dimensionalreconstructed image.
 11. The method of claim 10 wherein the secondcoordinate system is selected from a group including a Cartesiancoordinate system, a cone beam coordinate system and a coordinate systemassociated with a virtual acquisition system.
 12. The method of claim 9wherein the step of reconstructing includes the step of orienting saidthree-dimensional reconstructed tomosynthesis image to a selected plane.13. The method of claim 9 wherein the step of synthesizing includesprojecting at least one tomosynthesis slice image having a pixel spacingthat differs from a pixel spacing of a mammogram onto a mammogramcoordinate system.